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Distinguished members of the Joint Committee on Standards for Educational Evaluation: 

Thank you for this opportunity to respond to the February 1993 draft of the Program Evaluation 
Standards. My name is Charlene Rivera; I am a Senior Research Scientist at the George 
Washington University, and Director of the Evaluation Assistance Center East. I have worked 
in a range of college, university, and research and evaluation settings, and as a visiting scholar 
at ETS, worked on the nation-wide NAEP project; however, I will focus my testimony today 
on issues related to the K-12 educational system, and to the education of linguistically and 
culturally diverse students and their families. 



To set the context for this testimony, let me briefly explain the work of the Evaluation Assistance 
Center East and the constituencies it serves. The Evaluation Assistance Center East is one of two 
EACs funded by the Department of Education's Office of Bilingual Education and Minority 
Language Affairs (OBEMLA) to provide technical assistance on assessment and evaluation to 
state and local education agencies which receive federal funding to support bilingual education 
programs. Our work includes on-site and remote provision of technical assistance on a range of 
evaluation-related issues to state educational personnel and to teachers, coordinators, 
administrators, and evaluators, in local schools in 27 states, the District of Columbia, Puerto Rico, 
and the U.S. Virgin Islands. We have also worked togetlier with staff from the Office of 
Bilingual Education and Minority Language Affairs to develop materials to facilitate compliance 
with the relevant federal evaluation requirements. We also work extensively in the areas of 
language and academic assessment of students who are English language learners (often referred 
^ to as "limited English proficient" students). 
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Over 1 100 local projects are currently funded by OBEMLA; each of these projects is evaluated 
annually in keeping with the relevant federal evaluation requirements. Because we at the EAC 
East work directly with project directors-all of whose responsibilities include oversight of 
program evaluation, but many of whom have not received extensive training in evaluation-I will 
begin with a comment regarding the utility of the standards themselves to practitioners. 
Subsequentiy, I will comment on the adeq;V'Cy of six of the standards in accounting for and 
addressing issues in evaluation in culturally and linguistically educational environments. 

!• Utility of the Standards to Practitioners 

As noted in the introduction to the draft, the standards are intended to help a i^ange of audiences 
in understanding, judging, using evaluation. On the positive side, the document as it stands is 
weU written, and generally clear. However, there are still audiences for whom the standards 
themselves may be daunting merely by virtue of their size and number. In our work in the field, 
we encounter many project directors who, while well qualified to direct educational programs, 
have no experience with, and consequentiy have high anxiety about, guiding a program 
evaluation. I suspect that for such "reluctant audiences," a brief, even more user-friendly 
introduction to the standards might be useful. What 1 have in mind is a separate volume that 
would orient educational practitioners to good evaluation practice, introduce the main categories 
of the standards, aiid encourage them to examine the full set as a reference. 

Let me reiterate that the writing style of the standards is generally clear, and the use of examples 
make the document quite accessible. While changes in the standards themselves are not 
necessarily warranted, some people still react negatively to things they don't already understand, 
or that they think will be difficult to understand. Supplemental or auxiliary materials, whether 
developed under the direction of the Committee or developed by third parties, would be of great 
utility to many educational practitioners. 

2. Applicability to evaluation of bilingual education programs 

As a whole framework, the standards will certainly support and confirm good practices in the 
evaluation of bilingual education programs; while I suppose others may, in their testimony. 
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suggest changes in the organization of the framework, I see nothing in it that warrants alteration. 
Rather, I will respond to six (6) of the standards in light of the needs, rights, and experiences of 
students and families who are still in the process of learning English-groups which are large, 
growing, and will certainly be a major presence in U.S. education and society throughout the time 
period between the approval of this set of standards and the development of the next revision 
sometime in the next millennium. 

Utility Standard 1: Audience identification 

As an educator, I am happy to see that the first standards treated in the document relate to utility 
of evaluations, and that the very first of these deals with audience identification. In order to 
further ensure that some audiences which are all too often not identified are included, I would 
recommend that the needs and rights of language minority individuals and communities be 
addressed explicitly. While it might be argued that the need to attend to language minority 
audiences is addressed in the spirit of the standard, I feel strongly that this sizable population 
ought to be mentioned explicitly. Let me review a few figures with you to substantiate this 
point; the national demographic data leads me to conclude that all evaluations ought to pay 
specific attention to such real and potential audiences. 

Between the 1980 census and the 1990 census, the number of persons 5 years of age or older 
who reported themselves to be speakers of non-English languages increased by 38% from 23 
•million to nearly 32 million (Numbers and Needs, vol. 2, no. 4, July 1992). Based on estimates 
from NELS 88 data, the number of non-English language background children and youth (aged 
5 to 17) is nearly 14% (6.3 million) of those aged 5-17 (Numbers and Needs, vol. 2, no. 5, 
September 1992). According to OBEMLA, at least 2.3 million K-12 students have been 
identified by states as having "limited" proficiency in English (OBEMLA 1992); since under- 
identification is a greater problem than over-identification, English language learners in the 
nation's student body are more numerous than this figure-which already represents more than 
one in twenty of the nation's 40.5 million students- would suggest. A better estimati^ that is 
sometimes discussed is 3.3 million (Stanford Working Group, 1993). These figures suggest 
strongly that very few, if any, educational evaluations that are undertaken in the lifetime of the 
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1993 standards will not have audiences who speak languages other than English, and whose 
needs, rights, and perspectives will not be addressed if communication occurs solely in English. 

Specifically, I would suggest adding a guidv^,line to standard Ul: 

In identifying stakeholders, be certain that all audiences affected by the program and its 
evaluation are identified and consulted, especially those whose perspectives might be 
overlooked because of their gender, ethnicity, or language background. Evaluators should 
make use of translators in cases where the evaluator is unable to communicate in the 
language(s) of some audiences, for example, patents and/or students who are not 
proficient in English. 

Utility Standard 5: Report Clarity 

In relation to standard U5, Report Clarity, ensuring inclusion of linguistic minority audiences in 
considering the findings of evaluations is again crucial. None of us can clearly understand 
information reported to us in languages we have not mastered sufficiently for that task. I would 
emphasize that in relation to this standard, I am not referring only to evaluations of programs that 
specifically target English language learners, but to all evaluations having audiences for whom 
the English language itself is not the most comprehensible medium. Once again, though this 
consideration can be inferred from the intent of the standard, including a direct and explicit 
statement would be appropriate; such a statement might read: 

Evaluation results should be made available in the languages of all interested audiences 
through verbal explanation or translation at public hearings and tlirough translation of 
reports or executive summaries of reports into the language(s) of all audiences. 

Utility Standard 6: Report Dissemination 

Issues discussed in relation to this standard relate to "right to know" audiences and to planning 
for dissemination that does not exclude those who should be part of the evaluation process at the 
reporting and responding stage. Evaluators and those responsible for evaluations should be 
specifically encouraged to ensure that linguistic minority audiences are included in the definition 
of "right to know" audiences, and that dissemination to language minority commimities takes 



place in appropriate and approachable contexts. Beyond the issue of using audiences* languages 
in reporting, evaluators should seek to involve community members in advisory and/or 
dissemination capacities to ensure that social impediments to dissemination are overcome. 
Planning for such dissemination and making feedback possible is crucial. 

Propriety Standard 3: Rights of Human Subjects 

There is both a broad, general concern regarding this standard, and a more practical one Taking 
the latter first, language issues ought to be addressed in relation to the guidelines dealing with 
informed consent, namely F, G, H, and I. Communication with language minorities in general 
must be in an understandable medium; particular care must be taken when communicating with 
the home about issues of informed consent Even if a student from a non-English speaking 
background has become proficient in communicating and ieaming in English, one cannot assume 
tliat communications sent to the home in English will be understood. Evaluators should be 
encouraged to work with school personnel knowledgeable about the language backgrounds of 
students and parents in the community, and to communicate appropriately with them. 

I also want to make a more general point; given the added time that consulting with 
knowledgeable personnel might entail, one can imagine that monolingual evaluators, or 
multilingual evaluators who are not proficient in one or more of the languages represented by 
students and families in a given educational setting, might be tempted to exclude linguistically 
diverse students and families from evaluations. This is not an ethical or acceptable route to take, 
and I would encourage the Committee to consider adding a general inclusiveness standard 
indicating that all groups of students who will or could be affected by the results of an evaluation 
must be fully included. 

Propriety Standard 6: Disclosure of Findings 

This may ^dl start to sound familiar! Guideline H under the 'disclosure of findings' standards 
encourages evaluators to "[e]ncourage clients to provide all affected persons with information that 
is appropriate and timely, and that helps them to be enlightened contributors, consumers, critics, 
and observers." I would recommend the addition of wording encouraging them specifically to 

5 



ERIC 



6 



ensure that complete information is practically and linguistically accessible in comprehensible 
forms for all relevant audiences. 

Accuracy Standard 5: Valid Measurement 

The treatment of this standard, as might be expected, is long and detailed; instrument validity 
per se, however, is not treated in great detail, but rather is dealt with by referring readers to 
appropriate testing standards. While this is an economical way to deal with the issue, I would 
suggest shifting one statement listed among the "common errors" to the "guidelines" column, and 
expanding it to include cognizance of language issues in relation to validity of measurement in 
particular and of evaluations in general. Namely, I would suggest that "common error" H, 
"failing to consider characteristics of respondents that may affect their ability to understand or 
respond to a procedure as implemented," be changed to a positive guideline that enjoins 
evaluators to consider all relevant characteristics of respondents, including consideration of 
linguistic appropriateness of measures and procedures. Too often, particularly in educational 
settings, though certainly in other settings as well, students who are still learning English are 
either excluded from consideration or inappropriately assessed. In the former case, the validity 
of program evaluations is compromised because all participants' experiences are not accounted 
for; in the latter case, data validity is compromised because an extraneous variable—language 
proficiency-confounds the results. 

3* Inclusion and placement of references 

Finally, I would add a general comment on the format While including references at the end 
of each standard is helpful, the same publications appear repeatedly at the end of various 
standards. It might be more useful to include a general list of references at the end of the 
document; annotated references to articles or chapters within books that pertain directly to the 
standard in question might be useful at the end of each section. 

Let me close by thanking the Committee for its diligent work in revising the Program Evaluation 
Standards, and for its efforts to elicit feedback from a wide range of constituencies. As technical 
assistance providers, we appreciate having access to good materials to use in providing assistance 
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regarding evaluation. The 1993 Program Evaluation Standards will certainly benefit our efforts 
greatly. I hope that you will seriously consider the recommendations made in these remai'ks. 
If there is any way that I or my staff can assist you in addressing the needs of culturally and 
linguistically diverse constituencies to educational evaluations, we will be happy to do so. 

Thank you. 
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Postscript: In addition, CR pointed out that none of the illustrative cases included in the 
document dealt with educational programs serving English language learners; she 
recommended that one or more be included. 
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